1. Field of the Invention
A method for recording uncoded ratings by multiple evaluators on a single automatic machine scannable document, maintaining independence of ratings and accuracy of results and permitting utilization of automatic optical scanning equipment in ratings capture.
2. Description of the Prior Art
The introduction of large-scale standardized writing examinations has greatly increased the need for automatic machine grading. Because written examinations must first be hand-scored by multiple graders and because each grader's score must be completely independent from any other grader's score, grades often are recorded on a sheet of paper and then are key-entered. Unfortunately, key operator entry is prone to error and multiple error opportunities are introduced as identification and score data are entered for multiple reading. Key entry errors pose serious additional problems when separate subtest component such as Reading test scores and Mathematics test scores must be merged with Writing test scores. Attempts have been made to use machine-scannable documents to collect multiple reader-assigned scores, but these attempts have required either the sacrifice of grader independence, introducing a serious measurement bias, or they have required using individually coded score scales for each rater, introducing the potential for substantial clerical error.
Various attempts have been made to provide solutions to some of these problems. Specifically, Parmenter (U.S. Pat. No.2,373,498) discloses an answer sheet or sleeve 12 which slips over examination sheet 11 with the questions 10 imprinted thereon. Rugland discloses a self-scoring test device consisting of marking of the stations 14 which will alternatively show the correct answers as at 15A or incorrect answers as at 16A to the student as the test is actually marked.
Shaw (U.S. Pat. No. 3,203,116) is directed toward an electromechanical test scoring or grading sheet including slots 20/22 which cooperate with registry pens 26/28 respectively to ensure alignment of the completed test sheet 10 with the grading machine itself.
Parmenter (U.S. Pat. No. 3,266,049) discloses an examination paper in booklet form including an overlap answer column 16 corresponding to the questions sheets (page 12/14).
Sokolski (U.S. Pat. No. 3,900,961) simply shows a test sheet for use with an automatic machine grading including an answer control and selective answer column.
These efforts have focused on the design and use of answer documents either to provide immediate correct-answer information to examinees or to capture examinee responses on a machine scannable form. None of these efforts has responded to the need to capture through automatic machine scanning scores assigned by multiple graders ensuring grader independence, and tabulating and correlating with a high level of accuracy.